An Expressive and Efficient Language for Information Gathering on the Web

While network query engines make it possible to gather and combine data from multiple Web sources, these systems primarily focus on efficient query execution and do not solve some of the more complicated problems of online information gathering. Such problems require alternative types of control flow and better integration with the external world; the unique nature of the Web requires query plans be expressive enough to accommodate these demands. In this paper, we describe an information gathering plan language that is expressive and promotes efficient execution. Through its support for subplans, recursion, and a unique set of operators, the language allows plans that can interactively gather data over a series of pages, monitor remote sources, and asynchronously notify users of updates and results. We also present a execution system that efficiently implements the plan language using a dataflow-style executor capable of pipelining data between operators.